Amanda Caswell writes that Google DeepMind piloted the first double-blind evaluation of a proprietary frontier LLM, using confidential computing to keep Gemini 2.5 Flash Lite's model weights hidden from evaluators while keeping the test questions hidden from Google. The setup combines Google Cloud Confidential Space, an NVIDIA H100 Confidential GPU, and Intel TDX host memory encryption, with both sides' data transmitted over encrypted connections into an enclave where evaluation occurs without either party accessing the other's protected assets.
- Research cited in the technical report found benchmark leakage in roughly half of 31 models tested
- OpenMined's PySyft handles network controls, blocking evaluation code from making external connections
- The paper puts performance overhead at under 5%, citing legal agreements and code reviews as the real bottlenecks
- The Confidential Space guest OS is open source but relies on private signing keys, so individual builds can't be independently reproduced
- Researchers are already looking at H100 and B200 GPU clusters connected via encrypted links for models too large for a single GPU
- Research cited in the technical report found benchmark leakage in roughly half of 31 models tested
- OpenMined's PySyft handles network controls, blocking evaluation code from making external connections
- The paper puts performance overhead at under 5%, citing legal agreements and code reviews as the real bottlenecks
- The Confidential Space guest OS is open source but relies on private signing keys, so individual builds can't be independently reproduced
- Researchers are already looking at H100 and B200 GPU clusters connected via encrypted links for models too large for a single GPU
